What the Crowd Sources: A Protocol for a Contribution-Centred Systematic Literature Review of Data Crowdsourcing Research

Data crowdsourcing is the mobilization of large groups of contributors—often volunteers via the Internet—to collect and/or analyze data. Research on data crowdsourcing often prioritizes the data consumer or project sponsor. Significant gaps remain in understanding how to address design issues from the perspective of data crowdsourcing contributors. A systematic literature review is an ideal method for identifying gaps in how researchers conceptualize contributions in data crowdsourcing. This project presents a protocol for such a systematic literature review of data crowdsourcing. We will use the protocol to guide a subsequent systematic literature review and the construction of a data-information-knowledge-wisdom chart that identifies critical gaps and opportunities for research in data crowdsourcing systems

Leverage analysis: A method for locating points of influence in systemic design decisions

Many systemic design processes include the development and analysis of systems models that represent the issue(s) at hand. In causal loop diagram models, phenomena are graphed as nodes, with connections between them indicating a control relationship. Such models provide mechanisms for stakeholder collaboration, problem finding and generative insight and are powerful . These functions are valued in design thinking, but the potential of these models may yet be unfulfilled. We introduce the notion of “leverage measures” to systemic design, adapting techniques from social network analysis and systems dynamics to uncover key structures, relationships and latent leverage positions of modelled phenomena. We demonstrate their utility in a pilot study. By rethinking the logics of leverage, we make better arguments for change and find the place from which to move the world

Finding the emic in systemic design: Towards systemic ethnography

An under-emphasized but crucial variable of success in systemic design is the perspective through which a problem system is understood and from which interventions are conceptualized and implemented. While rooted in design (a consciously empathetic discipline; cf. Kimbell, 2011), it is easy for systemic designers to use research practices that may fail to capture and use the perspectives of their stakeholders. ftese approaches risk misrepresenting the stakeholders who contribute to projects and, in turn, they are a danger to the potential impact of these mis-researched problem systems. In this research, I propose an assessment framework to check whether a project effectively deploys research tools and processes that strengthen stakeholders’ perspectives, and I provide a proof of concept of this framework in use through hermeneutic case study analysis. Systemic design processes that are not executed with the direct and explicit engagement of stakeholders—to the extent of achieving an emic (from within) understanding of the system—may be flawed at their foundation. By fostering recognition of the importance of an emic perspective, and by providing a framework of principles, practices, and processes to accomplish systemic design with this perspective, I hope to ensure that systemic design processes are as accurate and valid as possible with respect to the stakeholders of the system. ftis is not to suggest that systemic design practice is “too etic” (from outside). In fact, with roots in design, systemic design is often deliberately emic. Systemic designers make use of designerly tools that help the researcher to build empathy with system stakeholders (e.g., soft systems methodology, critical systems heuristics, appreciative inquiry; Jones, 2014). ftey often seek to engage stakeholders in the systemic design process and include reflective analysis of what has been learned in order to assess where deeper engagement with the system is required (Ryan, 2014). ftat said, with the advent of crowdsourcing (the facilitated involvement of the general public in problem solving, usually using online tools; Lukyanenko & Parsons, 2012) and data science (the use of computational tools to analyze and understand large quantities of data; Šćepanović, 2018), data-driven methods may increasingly influence systemic design practice. One recent example sought input from hundreds of people to identify opportunities for change in Canadian post-secondary systems through an iterative online survey (Second Muse, Intel, & Vibrant Data, 2016). ftis data-driven direction is a powerful opportunity, of course, but it underscores the need to develop principles and best practices for assessing and supporting emic understanding as we gain more data from these tools. In the first phase of this research, I look to the principles and theorists of ethnography to develop a framework for assessing the emic/etic perspective of a given research project. Namely, Geertz’ “ftick Description: Toward an Interpretive fteory of Culture” (found in The Interpretation of Cultures, 1973, chapter 1) provides a foundation for the process of emic research in the form of four iterative steps: (1) acknowledge initial impressions; (2) capture speech, behaviours, events, and artifacts; (3) construct meaning; and (4) self-appraise sufficiency of capture and construction of meaning. Meanwhile, Creswell and Miller (2000) provide a set of five procedural principles for emic validity: (1) triangulation; (2) disconfirming evidence; (3) prolonged engagement; (4) member checking and collaboration; and (5) researcher reflexivity. Taken together, I generate a critical research framework which can be used to assess a given research project’s emic/etic perspective. In the second phase, I provide a proof-of-concept of this framework (and its theoretical underpinnings) via a case-based assessment of three systemic design projects. Case studies provide an effective venue for learning about the context- dependent manifestations of the phenomena being studied (Flyvbjerg, 2006). One of these case studies is one I have developed through my experience in participating and contributing to the development of the Canadian National Youth Leadership and Innovation Strategy framework, which convened hundreds of youth and youth-serving organizations in order to understand the youth leadership and innovation system in Canada (MaRS Studio Y, 2017). fte second and third case studies are those profiled by Ryan and Leung (2014). In each case, I use identify phenomena representing the practice of emic (or etic) understanding in the research orientation of the work, as acknowledged by the above framework. I examine the step-by-step procedure and any associated notes about the experience of the researchers and participants involved. In each step or experience, I look for evidence of the four steps of emic understanding or the six techniques of emic validation reported above. In order to interpret and analyze the chosen case studies, I turn to the methodology of phenomenological hermeneutics (Eberle, 2014, p. 196; cf. Wernet, 2014). Phenomenological hermeneutics are appropriate as I have access to the described phenomena of the systemic design projects captured by the chosen cases, but these phenomena are not explicitly captured with reference to emic or etic perspectives—thus some construction of the inherent emic or etic data is necessary in order to make judgments about the perspectives found in the projects. ftis hermeneutical analysis provides comparative evidence for the emic and etic perspectives used by the researchers in each case. It becomes possible to contrast and critique the principles, practices, and processes employed in each project in order to make a judgment about the project’s resulting emic/etic orientation. From these analyses, a metaphor emerges. Systemic design projects with etic orientations adopt an intensivist approach. Akin to intensive care in medicine, the systemic designers attempt to artificially suspend a system in a room. (Consider board room systems mapping as a trivial example of this practice.) Attempts are made to “get the whole system in the room”, but the system is therefore removed from its context. fte status of inaccessible elements of the system are guessed at and assumed, while other elements are placed in stasis and augmented by facilitation and technology. fte resulting interventions are spun up in this artificial space, but implemented in the system’s context—the systemic design team simply hopes that their assumptions hold and that the artificial suspension didn’t cause too much damage. System design projects with an emic orientation adopt an extensivist approach. fte designers themselves extend into the system. ftey sit with it for a while in order to acclimatize to its culture and learn its patterns. ftey interact with stakeholders and phenomena in context and capture these interactions as they are, as an ethnographer would. fte interventions they develop are (co-) created in place, built into the system’s real networks and activities. Of course, the challenge with these dueling approaches is that there are important trade-offs. fte extensivist approach takes time and personal investment. What’s more, the intensivist approach can have other valuable outputs: stakeholders of a system see one another and the parts of the system they interact with as a cohesive whole. fte result of this analysis, then, is not an obvious set of best practices. Instead, the emic/etic assessment framework can be used to judge how a research project effectively captures the perspectives of its stakeholders. It breaks down a project into components, each of which provides an intervention point for enhanced emic understanding. Finally, it provokes a reflective conversation, forcing us to ask ourselves where we can do better

Innovation Education

Education reform presents an opportunity to improve innovation education and, in turn, advance innovation capacity. I synthesize the framing and strategy of resources from provincial, national, international, and theoretical perspectives on innovation in order to develop a holistic model of innovation and a curricula for innovation education. Then, I use systemic design to model Newfoundland and Labrador’s current education system and to suggest strategies for reform to enable improvement in Newfoundland and Labrador’s innovation education. Finally, I explore how systemic reform in Newfoundland and Labrador may serve as a systems laboratory for reform efforts in other jurisdictions

Monitoring Natural Events Globally in Near Real-Time Using NASA's Open Web Services and Tools

Since 1960, NASA has been making global measurements of the Earth from a multitude of space-based missions, many of which can be useful for monitoring natural events. In recent years, these measurements have been made available in near real-time, making it possible to use them to also aid in managing the response to natural events. We present the challenges and ongoing solutions to using NASA satellite data for monitoring and managing these events

Give me the place to stand: Leverage analysis in systemic design

A key component of many systemic design processes is the development and analysis of systems models that represent the issue(s) at hand. A system is a collection of interdependent social, technological, and environmental phenomena. Models of systems often take the form of Causal Loop Diagrams (CLDs—sometimes referred to as influence diagrams) in which phenomena are graphed as nodes with connections between them indicating an influencing relationship. These visual modelling techniques provide systemic designers with a mechanism for stakeholder collaboration, problem finding, and generative insight (i.e., sticky note ideation makes everyone feel heard, appears democratic, and often results in emergent themes and ideas). These functions are valorized in design thinking, and they provide real value in garnering momentum and achieving common mental models in complex problems. They give systemic designers powerful resources for use in visual argument. However, while we believe these tools are useful, we also believe their true potential is unfulfilled. The properties of complex systems (and of how people engage with them) present a number of issues that introduce bias and chance into this process (Norman & Stappers, 2015). Given a model, systemic designers work through what they observe and interpret, engage in dialogue about what is important, and look for patterns (one category of which is archetypes, in which phenomena following certain patterns tend to produce similar emergent behaviours; Braun, 2002). While some principles and processes exist (see Jones, 2014), identifying leverage points and designing solutions tends to happen by “muddling through” a problem. This means solutions are developed and implemented in opportunistic form, through satisficing rather than optimizing (Norman & Stappers, 2015; see also Simon, 2008, chapter 2). Thus, we find a critical value gap: models are used in visual argument, but they could be used to augment those very arguments founded on evidence and logical relationship analysis. We propose the application of semi-quantitative analytics to systemic design models to go beyond visual argument, offering a powerful toolkit for: Comprehensive system mapping for complex sociotechnical systems (including the development of reference models that can inform synthesis/Gigamaps, or that can be used as their own arguments); Network-based analysis to uncover key structures, relationships, and latent leverage positions of modelled phenomena; Analytical mapping of problem systems and sorting out multicausality; A toolkit for cross-impact analysis between problematiques; and A “reality check” on strategic foresight proposals (by mapping temporal changes in networks and problematiques, we can better predict signal -> trend outcomes). With these analytics, models may be rethought in terms of the logics of leverage to reconcile this value gap. We introduce (or at least renew emphasis) on centrality analysis (metrics derived from social network analysis, evaluating the relative importance of mapped phenomena through measuring the structure of the directed graph made by the phenomena) and decomposition heuristics (algorithms derived from systems dynamics that analyze the directed graph structure to reveal the causal and loop hierarchy of modelled systems) in systemic design. To demonstrate the application of centrality analysis, we map the interconnectivity of the Sustainable Development Goals (SDGs) and their targets based on the work of Le Blanc (2015). By using metrics adopted from social network analysis, we are able to differentiate between goals and targets of differing levels of importance based on the structure of the map. Phenomena closeness (how proximate a given element is to the rest of the map) provides a ranked list of key indicators of change in the mapped system. Eigenvector (how well-connected an element is to other well-connected elements) analysis provide a ranked list of highly connective forces in the system: potential leverage points. These metrics therefore help identify which goals and targets to watch and which to intervene on the process of creating systemic change in the SDGs. To demonstrate the application of decomposition heuristics, we create a level partition (a hierarchy of causal structure of a map) and a loop inclusion graph (a hierarchy of feedback loop subsystems nested within one another) from feedback loops modelled in previous work on education systems change (Murphy, 2016). The level partition only decomposes the system into two levels, showing the strongly connected nature of the modelled phenomena in the system at hand. The loop inclusion graph, however, shows that certain feedback loops dominate the feedback loops they are contained within. Understanding—and intervening upon—these dominant loops should take precedence over their subsidiaries. The potential value in combining these tools should be clear. Decomposition heuristics can be used to break down the structure of modelled systems, making clear hierarchies and isolated systems within systems that sometimes disappear in the hairball complexity of these models. Likewise, centrality analytics can indicate key indicators, leverage points, bottlenecks, and other useful phenomena in the system. Taken together, isolated, dominant subsystems with high rankings on centrality measures tell systemic designers exactly where to stand in order to move their systems. The resolution of this value gap is particularly important as we see growth in the use of systemic design—and the technologies that support its practice. In order to develop models of systems that accurately represent the many stakeholders involved in the system, systemic designers must draw on diverse sources to collect and organize as much data as possible (Jones, 2014; Stroh, 2015). Fortunately, thanks to the development of recent technologies and practices such as crowdsourcing (the development of participatory systems that involve publics in a collaborative project, usually directed by a project owner; Lukyanenko & Parsons, 2012) and data science (a set of techniques and theories that help distill insight from data; Šćepanović, 2018), the collection and organization of large amounts of data will become ever easier. This brings us to an important paradox. Larger, more complex, data-driven models are likely more representative, as they capture more perspectives and nuances than simpler models. At the same time, larger, more complex models are harder to learn and understand (Rossi & Brinkkemper, 1996), and therefore they are also harder to use in the development of solutions. Thus, the tools we propose come at a crucial moment for leverage analysis in systemic design. Their advancement and provisioning could elevate the potential of the tools at the core of the discipline. With this careful rethinking of the logics of leverage, we might make better arguments for change, finding the place to stand from which to move the world

Systemic Strategy: Systemic design methods for complex systems change

Systemic design is a key practice for systems change (Jones, 2017). Systems change is a transdiscipline (Brown, Harris, & Russell, 2010) that focuses on fostering strategic changes across systems to bring about positive social gains at scale (Gopal & Kania, 2015). Many foundations (”Systems Change,” 2020), philanthropists (Walker, 2017), investors (”Systems Change: An Emerging Practice in Impact Investing,” 2019), NGOs (Banerjee et al., 2019), and governments (OECD, 2017) have taken an interest in systems change work in recent years. Systemic designers wield many tools and mindsets that support understanding and designing for effective interventions in complex systems (Jones, 2014). In this paper, we bridge the gap between a conventional tool of social change programming (Theories of Change) and systemic design methodologies in a combined approach for the design and evaluation of systemic strategies. Theories of Change (ToCs; and Theories of Action, ToAs) are conventional and fundamental tools in program design and evaluation (Mackinnon, 2006). These tools make explicit a team’s understanding of the system they are working within, externalize assumptions and biases, and help teams create shared mental models about the ways their change strategies should work (”Theory of Change: A Practical Tool,” 2004). They are commonly used in the design and development of change interventions, as they help develop and engage collaborators in a common mental model of the problem and potential solutions (Abercrombie, Boswell, & Thomasoo, 2018, p. 5). However, ToCs can also be reductive (Abercrombie et al., 2018, p. 5). They are usually linear and compact, able to be shared on a single slide or a single figure in a funding proposal. Systemic design principles suggest that this simple linearity is problematic: models that are so simple likely represent an idealized vision of the system, ignorant of the counterintuitive complexities that make these systems so hard to change. We argue that while ToCs have their role in communicating core ideas about a system and convincing funders to invest in an initiative, they should not be used to drive systems change strategy decisions. Such strategies will at best be flawed and fail; at worst, change initiatives using too-reductive a model may exacerbate the problems they aim to solve when they fail to account for system feedbacks (Abercrombie et al., 2018, p. 5). Systemic design offers alternative tools for would-be users of theories of change, such as Causal Loop Diagrams (CLDs; Kim, 1992). However, these tools are sometimes too complex, making it difficult to come to a consensus about what to do in a system. Ergo, both ToCs and CLDs are important tools for systems change. ToCs and their counterpart, ToAs, provide a framework for quickly and concretely explaining the narrative of systems change. CLDs help designers understand the complex system they’re working within and identify opportunities for change. In a recent paper, we propose a novel approach that bridges ToCs and systemic design in the creation of Systemic Theories of Change (SToCs; Murphy & Jones, 2020). There, we demonstrated that tools like leverage analysis—based on analyses of causal loop diagrams (Kim, 1992)—can help designers break down the complexity of systems models. Program designers and evaluators can use these tools to embrace complex systems dynamics, rather than ignore them. In this paper, we build upon those methods, bridging SToCs with strategic planning. Through a strategy “seeds,” trees,” and “forest” metaphor, we show how SToCs can be converted to conventional strategies and aligned with strategy maps (Kaplan & Norton, 2000), how multiple SToCs can be brought together to create holistic systems change strategies, and finally how systemic designers can establish and manage comprehensive monitoring and evaluation assessments of whether their systemic strategies are working. We demonstrate these concepts via a case study. Our ultimate aim is to converge the important work of program design and evaluation with the principles and methods of systemic design, enabling systems change strategies to be simultaneously more effective and understandable

Open social mapping participatory: Modeling of social systems

Open Social Mapping is an emerging paradigm for stakeholder engagement in systemic design projects. It combines actor mapping, network modelling and analysis, customer relationship management systems, and crowdsourcing in a method that allows stakeholders to map themselves within a system. Based on observations of some early examples of this tool and two case studies led by the authors, we describe some of the opportunities and challenges of Open Social Mapping. Open Social Maps re-center the stakeholder in the systemic design process, helping designers make data- driven decisions with real-time data while decentralizing systemic design by facilitating stakeholder access and agency to the design process. However, we must address issues of data collection and maintenance, privacy, power and privilege, bad actors, interoperability, and information quality for this tool to become mainstream